Autonomous drive user interface

ABSTRACT

A user interface for a vehicle, including a steering wheel for the vehicle, including a grip, a sensor operable to detect objects at a plurality of locations along the grip, and an illuminator operable to illuminate different portions of the grip, a processor in communication with the sensor, with the illuminator and with a controller of vehicle functions, and a non-transitory computer readable medium storing instructions which cause the processor: to identify, via the sensor, a location of a first object along the grip, to illuminate, via the illuminator, a portion of the grip, adjacent to the identified location, to further identify, via the sensor, a second object being at the illuminated portion of the grip, and to activate, via the controller, a vehicle function in response to the second object identified as being at the illuminated portion of the grip.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.14/551,096, entitled LIGHT-BASED CONTROLS IN A TOROIDAL STEERING WHEELand filed on Nov. 24, 2014, by inventors Gunnar Martin Fröjdh, SimonFellin, Thomas Eriksson, John Karlsson, Maria Hedin and RichardBerglind, the contents of which are hereby incorporated herein in theirentirety.

This application is also a continuation-in-part of U.S. application Ser.No. 14/555,731, entitled DOOR HANDLE WITH OPTICAL PROXIMITY SENSORS andfiled on Nov. 28, 2014, by inventors Sairam Iyer, Stefan Holmgren andPer Rosengren, the contents of which are hereby incorporated herein intheir entirety.

U.S. application Ser. No. 14/551,096 is a continuation of U.S.application Ser. No. 14/312,711, now U.S. Pat. No. 8,918,252, entitledLIGHT-BASED TOUCH CONTOLS ON A STEERING WHEEL and filed on Jun. 24, 2014by inventors Gunnar Martin Fröjdh, Simon Fellin, Thomas Eriksson, JohnKarlsson, Maria Hedin and Richard Berglind, the contents of which arehereby incorporated herein in their entirety.

U.S. application Ser. No. 14/312,711 is a continuation of U.S.application Ser. No. 14/088,458, now U.S. Pat. No. 8,775,023, entitledLIGHT-BASED TOUCH CONTOLS ON A STEERING WHEEL AND DASHBOARD and filed onNov. 25, 2013 by inventors Gunnar Martin Fröjdh, Simon Fellin, ThomasEriksson, John Karlsson, Maria Hedin and Richard Berglind, the contentsof which are hereby incorporated herein in their entirety.

U.S. application Ser. No. 14/088,458 is a non-provisional of U.S.Provisional Application No. 61/730,139 entitled LIGHT-BASED TOUCHCONTOLS ON A STEERING WHEEL AND DASHBOARD and filed on Nov. 27, 2012 byinventors Gunnar Martin Fröjdh, Thomas Eriksson, John Karlsson, MariaHedin and Richard Berglind, the contents of which are herebyincorporated herein in their entirety.

U.S. application Ser. No. 14/555,731 is a continuation-in-part of U.S.application Ser. No. 14/312,787 entitled OPTICAL PROXIMITY SENSORS andfiled on Jun. 24, 2014 by inventors Stefan Holmgren, Sairam Iyer,Richard Berglind, Karl Erik Patrik Nordström, Lars Sparf, Per Rosengren,Erik Rosengren, John Karlsson, Thomas Eriksson, Alexander Jubner, RemoBehdasht, Simon Fellin, Robin Åman and Joseph Shain, the contents ofwhich are hereby incorporated herein in their entirety.

U.S. application Ser. No. 14/555,731 is a continuation-in-part of U.S.application Ser. No. 14/311,366 entitled OPTICAL TOUCH SCREENS and filedon Jun. 23, 2014 by inventors Robert Pettersson, Per Rosengren, ErikRosengren, Stefan Holmgren, Lars Sparf, Richard Berglind, ThomasEriksson, Karl Erik Patrik Nordström, Gunnar Martin Fröjdh, Xiatao Wangand Remo Behdasht, the contents of which are hereby incorporated hereinin their entirety.

U.S. application Ser. No. 14/555,731 is a continuation-in-part of U.S.application Ser. No. 14/140,635 entitled LIGHT-BASED PROXIMITY DETECTIONSYSTEM AND USER INTERFACE and filed on Dec. 26, 2013 by inventors ThomasEriksson and Stefan Holmgren, the contents of which are herebyincorporated herein in their entirety.

U.S. application Ser. No. 14/312,787 is a continuation of InternationalApplication No. PCT/US14/40112 entitled OPTICAL PROXIMITY SENSORS andfiled on May 30, 2014 by inventors Stefan Holmgren, Sairam Iyer, RichardBerglind, Karl Erik Patrik Nordström, Lars Sparf, Per Rosengren, ErikRosengren, John Karlsson, Thomas Eriksson, Alexander Jubner, RemoBehdasht, Simon Fellin, Robin Åman and Joseph Shain, the contents ofwhich are hereby incorporated herein in their entirety.

International Application No. PCT/US14/40112 is claims priority benefitof U.S. Provisional Application No. 61/828,713 entitled OPTICAL TOUCHSCREEN SYSTEMS USING REFLECTED LIGHT and filed on May 30, 2013 byinventors Per Rosengren, Lars Sparf, Erik Rosengren and Thomas Eriksson;of U.S. Provisional Application No. 61/838,296 entitled OPTICAL GAMEACCESSORIES USING REFLECTED LIGHT and filed on Jun. 23, 2013 byinventors Per Rosengren, Lars Sparf, Erik Rosengren, Thomas Eriksson,Joseph Shain, Stefan Holmgren, John Karlsson and Remo Behdasht; of U.S.Provisional Application No. 61/846,089 entitled PROXIMITY SENSOR FORLAPTOP COMPUTER AND ASSOCIATED USER INTERFACE and filed on Jul. 15, 2013by inventors Richard Berglind, Thomas Eriksson, Simon Fellin, PerRosengren, Lars Sparf, Erik Rosengren, Joseph Shain, Stefan Holmgren,John Karlsson and Remo Behdasht; of U.S. Provisional Application No.61/929,992 entitled CLOUD GAMING USER INTERFACE filed on Jan. 22, 2014by inventors Thomas Eriksson, Stefan Holmgren, John Karlsson, RemoBehdasht, Erik Rosengren, Lars Sparf and Alexander Jubner; of U.S.Provisional Application No. 61/972,435 entitled OPTICAL TOUCH SCREENSYSTEMS and filed on Mar. 31, 2014 by inventors Sairam Iyer, Karl ErikPatrik Nordström, Lars Sparf, Per Rosengren, Erik Rosengren, ThomasEriksson, Alexander Jubner and Joseph Shain; and of U.S. ProvisionalApplication No. 61/986,341 entitled OPTICAL TOUCH SCREEN SYSTEMS andfiled on Apr. 30, 2014 by inventors Sairam Iyer, Karl Erik PatrikNordström, Lars Sparf, Per Rosengren, Erik Rosengren, Thomas Eriksson,Alexander Jubner and Joseph Shain, the contents of which are herebyincorporated herein in their entirety.

U.S. application Ser. No. 14/311,366 is a continuation of InternationalApplication No. PCT/US14/40579 entitled OPTICAL TOUCH SCREENS and filedon Jun. 3, 2014 by inventors Robert Pettersson, Per Rosengren, ErikRosengren, Stefan Holmgren, Lars Sparf, Richard Berglind, ThomasEriksson, Karl Erik Patrik Nordström, Gunnar Martin Fröjdh, Xiatao Wangand Remo Behdasht, the contents of which are hereby incorporated hereinin their entirety.

International Application No. PCT/US14/40579 claims priority benefit ofU.S. Provisional Application No. 61/830,671 entitled MULTI-TOUCH OPTICALTOUCH SCREENS WITHOUT GHOST POINTS and filed on Jun. 4, 2013 byinventors Erik Rosengren, Robert Pettersson, Lars Sparf and ThomasEriksson; of U.S. Provisional Application No. 61/833,161 entitledCIRCULAR MULTI-TOUCH OPTICAL TOUCH SCREENS and filed on Jun. 10, 2013 byinventors Richard Berglind, Erik Rosengren, Robert Pettersson, LarsSparf, Thomas Eriksson, Gunnar Martin Fröjdh and Xiatao Wang; of U.S.Provisional Application No. 61/911,915 entitled CIRCULAR MULTI-TOUCHOPTICAL TOUCH SCREENS and filed on Dec. 4, 2013 by inventors RichardBerglind, Erik Rosengren, Robert Pettersson, Lars Sparf, ThomasEriksson, Gunnar Martin Fröjdh and Xiatao Wang; of U.S. ProvisionalApplication No. 61/919,759 entitled OPTICAL TOUCH SCREENS WITHTOUCH-SENSITIVE BORDERS and filed on Dec. 22, 2013 by inventors RemoBehdasht, Erik Rosengren, Robert Pettersson, Lars Sparf and ThomasEriksson; of U.S. Provisional Application No. 61/923,775 entitledMULTI-TOUCH OPTICAL TOUCH SCREENS WITHOUT GHOST POINTS and filed on Jan.6, 2014 by inventors Per Rosengren, Stefan Holmgren, Erik Rosengren,Robert Pettersson, Lars Sparf and Thomas Eriksson; and of U.S.Provisional Application No. 61/950,868 entitled OPTICAL TOUCH SCREENSand filed on Mar. 11, 2014 by inventors Karl Erik Patrik Nordström, PerRosengren, Stefan Holmgren, Erik Rosengren, Robert Pettersson, LarsSparf and Thomas Eriksson, the contents of which are hereby incorporatedherein in their entirety.

FIELD OF THE INVENTION

The field of the present invention is steering wheel user interfaces forvehicles.

BACKGROUND OF THE INVENTION

Reference is made to FIG. 1, which is a simplified illustration of aprior art steering wheel. A steering wheel 400, shown in FIG. 1,includes a circular gripping member 401, one or more connecting members402-404 that connect the gripping member 401 to steering column 407, andbuttons 405 and 406 on connecting members 402 and 403 for controllingvarious devices in the vehicle. Connecting members 402-404, whichconnect gripping member 401 to steering column 407, are also referred toas spokes. In FIG. 1, button 405 is used to answer an incoming phonecall on the vehicle's BLUETOOTH speaker phone and button 406 hangs upthe call. BLUETOOTH is a trademark owned by the Bluetooth SIG ofKirkland, Wash., USA. Controls mounted in a steering wheel can beoperated comfortably and safely since the driver is able to control andoperate these controls without taking hands off the wheel or eyes offthe road.

Historically, the first button added to a steering wheel was a switch toactivate the car's electric horn. When cruise control systems wereintroduced, some automakers located the operating switches for thisfeature on the steering wheel as well. Today additional button controlsfor an audio system, a telephone and voice control system, a navigationsystem, a stereo system, and on board computer functions are commonlyplaced on the steering wheel.

U.S. Patent Publication No. 2012/0232751 A1 for PRESSURE SENSITIVESTEERING WHEEL CONTROLS describes adding pressure-sensitive controls tothe circular gripping member of the steering wheel. Pressure sensors arelocated at various locations along the perimeter of the gripping member,and different locations correspond to different controls. A control isactuated in response to application of pressure at a sensor location,e.g., by the user tightening his grip.

Prior art user interfaces associated with steering wheels, such as thebuttons and grips discussed hereinabove, associate a function with anabsolute position on the steering wheel. This is conceptually analogousto a touch-sensitive screen displaying icons where the user touches thelocation on the screen at which the icon is located to activate theicon. The concept of absolute positioning for user input goes back evenfurther: each key on a keyboard is positioned at an absolute positionwithin the keyboard. Similarly, early graphical user interfaces usinglight pens required the user to place the light pen onto a graphicdisplayed on the screen in order to activate a corresponding function.

In contrast to these user interfaces based on absolute positioning, thecomputer mouse introduced a user interface for controlling a cursorbased on relative positioning. Namely, the mouse cursor moves on thescreen in a direction that the mouse moves from point A to point B, butthis movement is not at all contingent on the actual coordinates—theabsolute positions—of points A and B. This shift from absolutepositioning to relative positioning frees the user from having to lookat, or be aware of, the location of the mouse on the table. The useronly has to control the direction in which the mouse moves on the table,which he can do without looking at the mouse. One of the objectives ofthe present invention is to provide a user interface for a driver basedon the relative positioning user interface model.

SUMMARY

The present disclosure relates to user interfaces for on board vehiclesystems, and teaches a user interface that does not require the user tobe aware of the location at which he is touching the steering wheel inorder to activate a function. The present disclosure teaches a robustvocabulary of user gestures that can be mapped to a wide variety ofapplications. The user gestures of the present disclosure are performedwith absolute confidence by a user, without the user looking at thesurface on which the gestures are performed. In certain embodiments ofthe invention the gestures are performed on the rim of a steering wheel.The nature of these gestures and the underlying hardware provided fordetecting these gestures enables each user interface gesture to beperformed by the user without any need for looking at the steeringwheel. Furthermore these gestures are entirely independent of how thesteering wheel is rotated at the time the gestures are performed.

There is thus provided in accordance with an embodiment of the presentinvention a steering wheel that identifies gestures performed on itssurface, including a circular gripping element including athumb-receiving notch disposed along its circumference, an array oflight-based proximity sensors, mounted in the gripping element, thatprojects light beams through the notch radially outward from thegripping element, and detects light beams reflected back into thegripping element by a moving object at or near the notch, and aprocessor, coupled with the proximity sensor array, for determiningpolar angles along the circumference of the gripping element occupied bythe object, responsive to light beams projected by the proximity sensorarray and reflected back by the object being detected by the proximitysensor array.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be more fully understood and appreciated fromthe following detailed description, taken in conjunction with thedrawings in which:

FIG. 1 is a simplified illustration of prior art steering wheel;

FIG. 2 is an exploded view of a steering wheel, in accordance with afirst embodiment of the present invention;

FIG. 3 is a cutaway view of a segment of the steering wheel of FIG. 2,in accordance with an embodiment of the present invention;

FIGS. 4 and 5 are exploded views of the steering wheel segmentillustrated in FIG. 3, in accordance with an embodiment of the presentinvention;

FIG. 6 is a simplified illustration of electronic components in thesteering wheel segment of FIG. 2 connected to a processor, in accordancewith an embodiment of the present invention;

FIG. 7 is a simplified illustration of a structure of light bafflesplaced upon the electronic components in FIG. 6, in accordance with anembodiment of the present invention;

FIGS. 8 and 9 are simplified illustrations of light beams detecting anobject, in accordance with an embodiment of the present invention;

FIG. 10 is simplified side view illustration of light beams projectedradially outward from a steering wheel, in accordance with an embodimentof the present invention;

FIG. 11 is a simplified illustration of communication between touchdetection firmware and multiple clients over a network, in accordancewith an embodiment of the present invention;

FIG. 12 is a simplified illustration of five basic gesture componentsused in a steering wheel user interface, in accordance with anembodiment of the present invention;

FIG. 13 is a flowchart of an exemplary vehicle user interface, inaccordance with an embodiment of the present invention;

FIG. 14 is a simplified illustration of user interface gesturesperformed on a steering wheel for an exemplary adaptive cruise controlfunction, in accordance with an embodiment of the present invention;

FIG. 15 is a simplified illustration of a multi-touch double-tap gestureand an exemplary user interface to activate an autonomous drive mode, inaccordance with an embodiment of the present invention;

FIG. 16 is a simplified illustration of a gesture and an exemplary userinterface for exiting the autonomous drive mode, in accordance with anembodiment of the present invention;

FIG. 17 is a simplified illustration showing how an incoming call isreceived, in accordance with an embodiment of the present invention;

FIG. 18 is a simplified illustration showing how to hang up a call, inaccordance with an embodiment of the present invention; and

FIG. 19 is a simplified illustration of a user interface for a parkassist function, in accordance with an embodiment of the presentinvention.

In the disclosure and figures, the following numbering scheme is used.Light transmitters are numbered in the 100's, light detectors arenumbered in the 200's, light guides and lenses are numbered in the300's, miscellaneous items are numbered in the 400's, light beams arenumbered in the 600's, and flow chart elements are numbered 1000-1100.Like numbered elements are similar but not necessarily identical.

The following tables catalog the numbered elements and list the figuresin which each numbered element appears.

Light Transmitters Element FIGS. 100 4, 5 101 6 102 6 105 8-10 106 6, 7107 6, 7 108 6, 7 109 6, 7

Light Detectors Element FIGS. 200 4 201 6, 8, 9 202 6, 8, 9 203 8, 9 2058, 9 206 8

Light Guides and Lenses Element FIGS. 300 2-7  301 8, 9 302 8, 9 303 8,9 304 8, 9 305 8-10

Miscellaneous Items Element FIGS. Description 400 1 steering wheel 401 1grip 402 1 right spoke 403 1 left spoke 404 1 bottom spoke 405 1 answerbutton 406 1 reject button 410 12, 14-19 steering wheel 411 2-5 steeringwheel frame 412 2-5 top cover 413 2-5 thumb notch 414 2-7 PCB 415 2-5, 7light baffle 416  3, 5 transparent cover section 417  3, 5 transparentcover section 418 12, 14, 15, 17, 18 finger 419 12, 14, 16-19 hand 42012, 15 steering wheel surface 421-424, 428, 12, 15 hand/finger movementdirections 431-434 425 12  clock icon 426 12  finger 430 14  double-tapgesture 436 14-16 Illumination 437 14  movement of illumination 438 14,17 tap gesture 440  6, 11 Processor 441-443 11  network client 444 11 message bus

Light Beams Element FIGS. Description 601 10 light beam 602 3, 8-10light beam 603 8, 9 light beam 604 8, 9 light beam

Flow Chart Stages Element FIGS. Description 1001-1005 13 vehicleapplication state 1010-1019 13 vehicle application action

DETAILED DESCRIPTION

Aspects of the present disclosure relate to light-based touch controlsthat allow a driver to keep his hands on a steering element whileoperating peripheral electronic devices and automated features in avehicle.

According to a first embodiment of the invention, a steering wheel isprovided with a touch sensitive strip disposed along the entirecircumference of the steering wheel. In order to facilitate locating thestrip, it is disposed in a thumb receiving notch or groove that isetched or otherwise formed along the circumference of the steeringwheel. In addition to a touch sensor, there is also a visible-lightilluminator behind or around the touch sensitive strip that is used toindicate the state of the user interface to the user, and also indicatewhere certain tap gestures should be performed.

A user interface for this steering wheel is designed to be independentof the rotation of the steering wheel. Sweep gestures are clockwise andcounter-clockwise so that they are independent of rotation of the wheel.A function is activated in response to a gesture, such as a double-tap,performed anywhere along the circumference of the wheel. The activationof some functions places the user interface into a state in which one ormore additional functions can be selectively activated. In order toactivate these additional functions, the touch location at which theinitial gesture was performed is illuminated and subsequent gestures areperformed in relation to the illuminated portion of the wheel. When aportion of the wheel is thus illuminated, and the driver slides his handalong the steering wheel grip, the illuminated portion of the steeringwheel follows the hand so that the hand is always next to the locationfor performing subsequent gestures. Similarly, when the user switcheshands gripping the steering wheel, the illumination jumps to the newlygripped part of the wheel.

Reference is made to FIG. 2, which is an exploded view of a steeringwheel, in accordance with a first embodiment of the present invention.Elements of this steering wheel include steering wheel frame 411, PCB414, an array of lenses 300, a light baffle structure 415, and asteering wheel top cover 412. A thumb-receiving notch 413 is disposedwithin steering wheel cover 412.

Reference is made to FIG. 3, which is a cutaway view of a segment of thesteering wheel of FIG. 2, in accordance with an embodiment of thepresent invention. Thumb-receiving notch 413 is illustrated in FIG. 3.Two light transmissive portions of cover 412 are also shown in FIG. 3. Afirst light transmissive portion 417 forms the side wall ofthumb-receiving notch 413. Light beams traveling into and out of thisportion provide touch detection and proximity detection, as explainedbelow. Three touch detection light beams 602 are shown directed radiallyoutward from the steering wheel gripping element. The second lighttransmissive portion 416 forms a floor of thumb-receiving notch 413, andis used for visible illumination indicating a state of the userinterface to the driver, and at which location the driver should performadditional user interface gestures.

Reference is made to FIGS. 4 and 5, which are exploded views of thesteering wheel segment illustrated in FIG. 3, in accordance with anembodiment of the present invention. As shown in FIG. 4, two concentricrows of elements are mounted on PCB 414. Namely, an inner row of lightdetectors 200 and an outer row of light emitters 100. Light from theemitters enters lenses 300 through which it is re-directed out of thesteering wheel through light transmissive portion 417 as light beams602, illustrated in FIGS. 3 and 8-10. An object such as a thumb placedin notch 413 reflects the light back through portion 417 and lenses 300onto one or more of the light detectors 200, thereby providing touchdetection, as illustrated in FIGS. 8 and 9. Similarly, an object such asa user's hand placed along the outer rim of the steering wheel outsidenotch 413 and opposite light transmissive portion 417 also reflects thelight back through portion 417 and lenses 300 onto one or more of thelight detectors 200, thereby providing proximity detection.

FIG. 5 shows an exploded view from below of the steering wheel segmentillustrated in FIG. 3, in accordance with an embodiment of the presentinvention.

Reference is made to FIG. 6, which is a simplified illustration ofelectronic components in the steering wheel segment of FIG. 2 connectedto a processor, in accordance with an embodiment of the presentinvention. FIG. 6 shows processor 440 connected to PCB 414 on whichthree concentric arrangements of light elements are mounted, namely, aninner circular arrangement of inward facing light detectors, includingdetectors 201 and 202; a middle circular arrangement of inward facinglight emitters, including emitters 101 and 102; and an outer circulararrangement of outward facing light emitters 105-108. The inward facinglight emitters are used for touch and proximity detection and typicallyemit light in the near infrared range. Processor 440 controls activationof the emitters and detectors, and detects gestures performed on thesteering wheel based on these activations and based on the outputs ofthe detectors.

The outward facing light emitters are used to provide visual indicationsto the user by illuminating light transmissive portion 416 of thesteering wheel cover, and emit light in the visible range. Lenses 300are described in assignee's co-pending application U.S. Ser. No.14/555,731, entitled DOOR HANDLE WITH OPTICAL PROXIMITY SENSORS, thecontents of which are incorporated herein in their entirety byreference.

Reference is made to FIG. 7, which is a simplified illustration of astructure of light baffles placed upon the electronic components in FIG.6, in accordance with an embodiment of the present invention. FIG. 7shows PCB 414 and lenses 300 of FIG. 6, but with baffle structure 415placed above the mounted light elements.

Reference is made to FIGS. 8 and 9, which are simplified illustrationsof light beams detecting an object, in accordance with an embodiment ofthe present invention. FIGS. 8 and 9 show a light path used to detect anobject. Shown in FIGS. 8 and 9 are individual lens structures 301-305.Each lens structure serves a respective opposite emitter and twodetectors, one to the left of the emitter and one to the right of theemitter. Thus, for example, lens structure 305 serves emitter 105 anddetectors 205 and 206. In addition each detector is served by two lensstructures; e.g., detector 205 receives reflected light from lensstructures 304 and 305. In the example shown in FIGS. 8 and 9, lightfrom emitter 105 is reflected by an object (not shown) into lensstructure 303 and onto detector 203. Three segments of the detectedlight are indicated in FIGS. 8 and 9; namely, light beam 602 projectedoutward from lens structure 305 and radially outward of the steeringwheel, light beam 603 reflected by the object into lens structure 303,and light beam 604 directed by lens structure 303 onto detector 203.

Reference is made to FIG. 10, which is simplified side view illustrationof light beams projected radially outward from a steering wheel, inaccordance with an embodiment of the present invention. FIG. 10 shows acutaway side view of the light path illustrated in FIGS. 8 and 9. FIG.10 shows light beam 601 from emitter 105 entering lens structure 305,where it is redirected outward as light beam 602.

Methods for determining two-dimensional coordinates of an objectdetected by the disclosed proximity sensor array are described inassignee's co-pending application Ser. No. 14/312,787, entitled OPTICALPROXIMITY SENSORS, and U.S. Ser. No. 14/555,731, entitled DOOR HANDLEWITH OPTICAL PROXIMITY SENSORS, both incorporated herein in theirentireties by reference. Because the present application is for asteering wheel and the proximity sensor array is arranged along anarc-shaped grip of the steering wheel, the determined coordinates arepolar coordinates, including a polar angle and a radial coordinate. Thepolar angle corresponds to a coordinate along the proximity sensorarray, which in application Ser. Nos. 14/312,787 and 14/555,731 isdescribed as an x-axis coordinate. The radial coordinate corresponds toa distance from the proximity sensor array, which in application Ser.Nos. 14/312,787 and 14/555,731 is described as a y-axis coordinate.

Discussion now turns to the firmware and software used to detect andinterpret user gestures. There are five basic gesture components thatare detected by the hardware and low level drivers: (i) Thumb-Tap, (ii)Thumb-Glide, (iii) Thumb-Long-Press, (iv) Grab and (v) Rim-Tap. Thesecomponents are emitted on the network as they are detected, and are usedby higher level software to assemble more complex gestures such asdouble-taps. Application software interprets these gestures as inputcommands. In some embodiments of the invention multiple clientapplications are connected via a network to the detector firmware. Thefirmware sends information for each detected gesture component over thenetwork, and a client application translates that information intocommands and/or constructs compound gestures from multiple gesturecomponents.

Reference is made to FIG. 11, which is a simplified illustration ofcommunication between touch detection firmware and multiple clients overa network, in accordance with an embodiment of the present invention.FIG. 11 shows an exemplary network architecture in which processor 440sends detected gesture components over message bus 444, e.g., using theMessage Queue Telemetry Transport (MQTT) messaging protocol on top ofthe TCP/IP protocol, to connected clients 441-443.

The five basic gesture components are categorized according to whetherthey are performed by a large object (hand) or small object (thumb), andwhether the nature of the gesture component is discrete or continuous,as presented in the table below.

Component Description Object Type Thumb-Tap Tap thumb on steering SmallDiscrete wheel rim Thumb-Glide Glide thumb along Small Continuoussteering wheel rim Thumb-Long- Hold thumb on steering Small ContinuousPress wheel rim Grab Grab hold of steering Large Continuous wheel rimRim-Tap Tap hand on steering Large Discrete wheel rimThese gesture components are alternatively referred to as follows.

Component Alternative Name Thumb-Tap small-object tap Thumb-Glidesmall-object glide Thumb-Long-Press small-object touch-and-hold Grablarge-object grab Rim-Tap large-object tap

The parameters are the same for all gesture components; namely, timestamp, start angle (min_angle), end angle (max_angle), center angle(angle) and state.

The angle parameters refer to a polar angle along the steering wheel atwhich the object is detected. Because of the object's size, there is afirst polar angle at which the object begins (start angle) and a lastpolar angle at which the object ends (end angle). The midpoint betweenthe start and end angles (center angle) is used as the object's polarangle. The start and end angles are useful for determining the size of adetected object.

The state parameter takes on three values: RECOGNIZED, UPDATED andENDED. The ENDED state is applied to all discrete gesture components,and also when a continuous gesture component ends. The RECOGNIZED andUPDATED states are only applied to continuous gesture components. TheRECOGNIZED state is applied when a continuous gesture component is firstdetected. The UPDATED state is applied during the course of a continuousgesture component.

The discrete gesture components, Thumb-Tap and Rim-Tap, are emitted tothe clients after they happen, and then only one message is sent for thegesture component. They are only sent with the state ENDED.

The continuous gesture components, Thumb-Glide, Thumb-Long-Press andGrab, are emitted to the clients intermittently from the instant thatthey are recognized until they end when the hand or finger leaves therim. When they are first recognized, they are sent to the network withthe state RECOGNIZED. With a configurable interval, the gesturecomponent is reported to the network with new parameters and the stateUPDATED. When the gesture component ends, the gesture component is sentwith the state ENDED.

Reference is made to FIG. 12, which is a simplified illustration of fivebasic gesture components used in a steering wheel user interface, inaccordance with an embodiment of the present invention. FIG. 12 showsthe five gesture components performed by thumb 418 and hand 419 onsteering wheel 410. Some gesture components are illustrated both fromabove and from the side. When illustrated from the side, thumb 418 isshown interacting with steering wheel surface 420.

A Thumb-Tap gesture component is generated when a small object touchesthe rim (or gets very close) and then is lifted from the rim within ashort period. This period is configurable, but typically it is 100-200ms. FIG. 12 shows the Thumb-Tap gesture component from above and fromthe side, and illustrates the movement of thumb 418 by arrows 421.

A Rim-Tap gesture component is the same as a Thumb-Tap, but for a largeobject such as a hand. FIG. 12 shows the Rim-Tap gesture component fromthe side and illustrates the movement of hand 419 by arrows 424.

A Thumb-Glide gesture component is generated when a small object touchesthe rim and moves at least a certain threshold distance along the rim.That distance is configurable. When it continues to move, UPDATEmessages are sent when the object has moved a certain distance, alsoconfigurable. FIG. 12 shows the Thumb-Glide gesture component from aboveand from the side, and illustrates the movement of thumb 418 by arrows422 and 423.

A Grab gesture component is the same as a Thumb-Glide gesture component,but for a large object touching the rim, and with the difference thatthe Grab gesture component does not have to move to be reported on thenetwork. When the hand has been on the rim for a certain time threshold,the Grab gesture component is recognized and messages are intermittentlysent to the network. FIG. 12 shows the Grab gesture component from aboveby showing hand 419 gripping steering wheel 410.

A Thumb-Long-Press gesture component is generated when a small object ispresent, and not moving, on the rim. When the small object has beenpresent for a certain time, messages are sent intermittently to thenetwork about the gesture component. If the object starts moving, theThumb-Long-Press gesture component is ended and a Thumb-Glide gesturecomponent is started instead. FIG. 12 shows the Thumb-Long-Press gesturecomponent from above and from the side. Clock icon 425 indicates thetime threshold required to distinguish this gesture component from aThumb-Tap.

As mentioned above, gesture components are combined into compound userinterface gestures. In some cases, environment conditions at the gesturelocation are combined with the gesture component to define a gesture.For example, a Thumb-Tap gesture performed at one end of an illuminatedportion of the rim is translated into a first command, and a Thumb-Tapgesture performed at the other end of the illuminated portion of the rimis translated into a second command. The following table lists thedifferent gestures and compound gestures in the steering wheel userinterface, the gesture components that make up each gesture, additionalgesture parameters, and example context and commands for each gesture.

Gesture Additional Example Example Gesture Components Parameters ContextCommand Tap inside Thumb-Tap Thumb-tap Cruise Increase or notchperformed at top or control is decrease cruise bottom of illuminatedactive control speed in portion of illuminated 5 mph segment of steeringincrements wheel Tap on Rim-Tap During Reactivate phone steering phonecall interaction, e.g., wheel outer when phone call rim is active forset period of time. Enables hanging up the phone call with a clockwiseswipe gesture Single Two Thumb- Thumb-taps have Vehicle is in Activatecruise object Taps different time stamps, motion control and double-tapsimilar center illuminate inside notch angles location of double-tapSingle Two Rim-Taps Side of steering Car is not Activate Park objectwheel rim (left or moving, and Assist to park on double-tap right) atwhich Park Assist left or right side on steering double-tap is icon isof car, based on wheel outer performed displayed on tapped side of rimHUD rim Multi-touch Two Thumb- Thumb-taps have Autonomous Activatedouble-tap Taps similar time stamps, drive is not autonomous insidenotch different center active drive angles Extended Thumb-Long-Thumb-long-press Cruise Increase or touch inside Press performed at topor control is decrease cruise notch bottom of illuminated active controlspeed in portion of illuminated 1 mph segment of steering incrementswheel Grab Grab Autonomous Deactivate drive is autonomous active drive,and enter cruise control mode Swipe Thumb-Glide clockwise/counter-Cruise Increase or clockwise control is decrease distance active fromforward car in cruise control mode Radial swipe Thumb-Glide Thumb-glidedata Cruise Open cruise structures have control is control menu onsimilar center angles active HUD and different radial coordinates SlideGrab Grab data structures Portion of Move illumination have differenttime steering to new hand stamps and different wheel is location (followcenter angles selectively slide movement) illuminated Switch Grab Grabdata structures Portion of Move illumination hands have different timesteering to new hand stamps and different wheel is location (jump tocenter angles selectively other side of illuminated wheel)

Reference is made to FIG. 13, which is a flowchart of an exemplaryvehicle user interface, in accordance with an embodiment of the presentinvention. The flowchart illustrates the different application states,different commands within each state, and the gestures used to issuethose commands. The details of the gestures are illustrated in FIGS.14-19. In some embodiments a heads-up display (HUD) is provided.

The flowchart of FIG. 13 illustrates a highway scenario that includesthree driving modes: Normal Drive 1001, Adaptive Cruise Control 1002 andAutonomous Drive 1003. In Normal Drive mode, the driver steers thevehicle and controls its speed. In Adaptive Cruise Control mode thedriver steers the vehicle but the vehicle's speed, its distance from thenext vehicle on the road, and other parameters are controlled by thevehicle. In Autonomous Drive mode the vehicle is driven and steeredautomatically without driver input.

The user enters Adaptive Cruise Control mode from Normal Drive mode byperforming a double-tap gesture. The user enters Autonomous Drive modefrom Normal Drive mode and from Adaptive Cruise Control mode byperforming a multi-touch double-tap gesture. These gestures aredescribed below. In order to alert the driver that Autonomous Drive modewill begin shortly, the steering wheel is illuminated with anillumination pattern that indicates a countdown until Autonomous Driveis activated.

The user exits Adaptive Cruise Control mode by performing a double-tapgesture that opens a menu on the HUD for changing the mode 1015 ofcruise control. The user performs clockwise or counter-clockwise swipegestures to scroll through the different modes on the HUD, and performsa single-tap gesture to select the displayed mode. One of the modes isExit ACC 1018, and selecting this mode exits Adaptive cruise Control.Another mode configures the cruise control application to follow theroad signage 1019.

The user exits Autonomous Drive mode 1013 by grabbing the rim of thesteering wheel. In order to alert the driver that Autonomous Drive modeis about to exit, the steering wheel is illuminated with an illuminationpattern that indicates a countdown until Autonomous Drive isdeactivated. Upon exiting Autonomous Drive mode, the vehicle entersAdaptive Cruise Control mode.

In Adaptive Cruise Control mode 1002 the user adjusts a distance 1016between the vehicle and the vehicle directly in front of it, byperforming a clockwise or counter-clockwise swipe gesture. The useradjusts the speed of the vehicle by performing either a tap gesture oran extended touch gesture. When the vehicle enters Adaptive CruiseControl mode 1002 a segment of the steering wheel is illuminated. A tapgesture or extended touch gesture at one end of the illuminated segmentincreases the vehicle speed, and a tap gesture or extended touch gestureat the other end of illuminated segment decreases the vehicle speed.

A voice control state 1004 can be entered from Normal Drive mode andAdaptive Cruise Control mode. In this state, the user can initiate aphone call by saying “call” and the name of a contact from his phone'scontact list. Once the call has been connected, the user can hang up1010 by performing a clockwise swipe gesture. The user can also adjustthe volume 1011 by saying the word “volume” and then performing acounter-clockwise swipe gesture to raise the volume, or a clockwiseswipe gesture to lower the volume.

When an incoming phone call 1005 is received, the user can answer thecall 1012 by performing a counter-clockwise swipe gesture, or declinethe call 1012 by performing a clockwise swipe gesture.

Reference is made to FIG. 14, which is a simplified illustration of userinterface gestures performed on a steering wheel for an exemplaryadaptive cruise control function, in accordance with an embodiment ofthe present invention. In order to enter Adaptive Cruise Control modefrom Normal Drive mode the user performs a single object double-tapgesture. Namely, the user taps twice with his thumb on the thumb notchin the steering wheel. This gesture is illustrated in drawing (a) inFIG. 14, showing steering wheel 410, hand 419 gripping steering wheel410, and double-tap gesture 430. The present invention enables the userto perform the double-tap gesture 430 at any location along theperimeter of steering wheel 410.

When Adaptive Cruise Control is active the user has four options;namely, adjust cruise control speed, adjust the distance between thevehicle and the vehicle ahead, open an adaptive cruise control menu, andactivate Autonomous Drive mode. As mentioned above Adaptive CruiseControl is activated when the user taps twice with his thumb in thesteering wheel thumb notch. The location of these taps is subsequentlyilluminated to indicate to the user where to perform future gestures.This is illustrated in drawing (b) in FIG. 14, showing illuminatedsegment 436 of the steering wheel 410 at the location at whichdouble-tap 430 was performed. Thus, to increase the cruise control speedthe user performs a gesture, e.g. a single-tap, above the illuminatedportion. This is illustrated in drawing (d) in FIG. 14 showing tapgesture 438 at the counter-clockwise edge of illuminated portion 436.The “+” indicates that this gesture increases the speed of the vehicle.Drawing (e) in FIG. 14 shows gesture 438 performed at the clockwise endof illuminated portion 436, and the “−” indicates that the gesturedecreases the speed of the vehicle.

If the user slides his hand 419 along steering wheel 410, theilluminated portion 436 moves with the hand so that the user's thumb isalways next to the illuminated portion of the steering wheel. This isillustrated in drawing (c) in FIG. 14, in which hand 419 grippingsteering wheel 410 slides clockwise as indicated by arrow 428, andilluminated portion 436 also slides in the same direction as indicatedby arrow 437.

In some embodiments the cruise control speed is also adjusted inresponse to extended touch gestures above and below the illuminatedportion of the steering wheel. For example, the speed is adjusted by 5km/h in response to a tap gesture, and is adjusted by 1 km/h in responseto an extended touch gesture.

In order to increase or decrease the distance between the vehicle andthe vehicle in front of it on the road, the user performs clockwise andcounter-clockwise swipe gestures. These are illustrated in drawings (f)and (g) in FIG. 14. Drawing (f) illustrates a counter-clockwise gesture431 to increase the distance between vehicles, and drawing (g)illustrates a clockwise gesture 432 to decrease the distance betweenvehicles.

In order to change the mode of Adaptive Cruise Control the user performsa radial swipe gesture with his thumb across the width of the steeringwheel thumb notch. This is illustrated in drawings (h) and (i) in FIG.14. Drawing (h) illustrates swipe gesture 433 that moves outward acrossthe width of illuminated portion 436. Drawing (i) illustrates swipegesture 434 that moves inward across the width of illuminated portion436. Either gesture causes the HUD to present a mode option forselection. The user performs a single-tap gesture with his thumb in thesteering wheel notch to accept the displayed mode. The mode displayed inthe HUD is changed in response to a swipe gesture. For example, a firstmode is to follow road signage. If the user performs a single-tap whenthis mode is displayed on the HUD, a Follow Road Signage mode isactivated. If the user swipes clockwise or counter-clockwise, a next orprevious mode is displayed such as exit Adaptive Cruise Control. Theuser performs a single-tap to activate this mode. If no interaction fromthe user is received within a fixed amount of time, such as 5 seconds,then the change mode user interface is deactivated.

Reference is made to FIG. 15, which is a simplified illustration of amulti-touch double-tap gesture and an exemplary user interface toactivate an autonomous drive mode, in accordance with an embodiment ofthe present invention. Drawing (a) in FIG. 15 illustrates two fingers,418 and 426, simultaneously tapping at two locations on steering wheel410. The upper part of this drawing is a view from above, and the lowerpart of this drawing is a view from the side of each of the fingers 418and 426. The tap gesture is a brief down and up gesture illustrated byarrows 421 and 428 touching surface 420 of the steering wheel.

Once the user performs this multi-touch double-tap gesture, a series oflocations on the steering wheel are sequentially illuminated over timeto indicate a countdown until Autonomous Drive is activated, asillustrated in drawings (b) and (c). For example, viewing the uprightsteering wheel as a clock, drawing (b) illustrates a sequence ofilluminations that begins with (i) the 2:30 and 9:30 clock positionsindicated by a 1; followed by (ii) the 1:30 and 10:30 clock positionsindicated by 2; followed by (iii) the 12:30 and 11:30 clock positionsindicated by 3. Drawing (c) illustrates finally illuminating the 12o'clock position indicated by the word “Go” to inform the user thatAutonomous Drive is activated and the user can safely take his hands offthe wheel.

In order to exit Autonomous Drive mode and enter Adaptive Cruise Controlmode, the user grabs the steering wheel. Reference is made to FIG. 16,which is a simplified illustration of a gesture and an exemplary userinterface for exiting autonomous drive mode, in accordance with anembodiment of the present invention. FIG. 16 shows two hands 419gripping steering wheel 410, in accordance with an embodiment of thepresent invention. A series of locations on the steering wheel is thensequentially illuminated to indicate that Autonomous Drive mode is aboutto be the deactivated. For example, drawing (a) illustrates a sequenceof illuminations that begins with (i) the 11:30 and 12:30 clockpositions indicated by a 1; followed by (ii) the 10:30 and 1:30 clockpositions indicated by 2; followed by (iii) the 9:30 and 2:30 clockpositions indicated by 3. When Autonomous Drive mode is deactivated thevehicle enters Adaptive Cruise Control mode, and a portion 436 ofsteering wheel 410 next to one of the hands 419 gripping the steeringwheel is illuminated, as illustrated in drawing (b) of FIG. 16, and asdiscussed above with reference to FIG. 14.

In both Normal Drive mode and Adaptive Cruise Control mode the user canenable voice activated controls by tapping twice on the outer rim of thesteering wheel. When voice-activated controls are enabled the userdisables these controls by repeating the same double-tap gesture.

Two voice-activated controls are illustrated in FIG. 13: placing a phonecall and enabling volume adjustments. To place a phone call the usersays “call” and the name of the person to call, e.g., “call Mom”. Inorder to hang up the call the user performs a swipe gesture along thethumb notch in the steering wheel. To adjust the volume of a call or thestereo system, the user says the word “volume” and then adjusts thevolume up or down by swiping clockwise or counter-clockwise along thethumb notch in the steering wheel.

Reference is made to FIG. 17, which is a simplified illustration showinghow an incoming call is received, in accordance with an embodiment ofthe present invention. FIG. 17 shows that when an incoming call isreceived, the user answers or declines the call by swiping finger 418clockwise or counter-clockwise along the thumb notch of the steeringwheel, e.g. swipe counter-clockwise to accept the call and swipeclockwise to reject the call.

Reference is made to FIG. 18, which is a simplified illustration showinghow to hang up a call, in accordance with an embodiment of the presentinvention. The gesture to hang up a call is a clockwise swipe gesture.However, when a call has been active for a certain amount of time, thesystem ignores clockwise swipe gestures so that the user does notinadvertently hang up the call. In order to hang up the call, the userfirst taps the outer rim of the steering wheel, as shown by hand 419, toindicate that the system should respond to the next swipe gesture,followed by a clockwise swipe gesture by finger 418 to hang up the call.

In a city scenario the user interface provides a park assist functionthat automatically parks the car without the user's intervention.Reference is made to FIG. 19, which is a simplified illustration of auser interface for a park assist function, in accordance with anembodiment of the present invention. When the vehicle is moving at lessthan 30 km/h, the Park Assist function begins automatically scanning foravailable parking spaces. In addition, a faded Park Assist icon appearson the HUD, as illustrated in drawing (a) of FIG. 19. As the car furtherslows down, this icon becomes bolder until the car has stopped moving,as illustrated in drawings (b) and (c) of FIG. 19. The HUD then presentsinformation about available parking spots; e.g. whether the vehicle canfit into that spot. The user performs a double-tap on the outer rim ofthe steering wheel, as illustrated in drawing (d), by hand 419 to beginthe automated parking. To indicate to the Park Assist function that theparking space is on the left side of the car, the user performs thisdouble-tap on of the left half of steering wheel rim. To indicate to thePark Assist function that the parking space is on the right side of thecar, the user performs this double-tap on of the right half of steeringwheel rim.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. It will, however,be evident that various modifications and changes may be made to thespecific exemplary embodiments without departing from the broader spiritand scope of the invention. In particular, sensors other than opticalsensors may be used to implement the user interface, inter aliacapacitive sensors disposed along the circumference of the steeringwheel, and cameras that captured images of the steering wheel.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

1-21. (canceled)
 22. A user interface for a vehicle, comprising: asteering wheel for the vehicle, comprising: a grip; a sensor operable todetect objects at a plurality of locations along said grip; and anilluminator operable to illuminate different portions of said grip; aprocessor in communication with said sensor, with said illuminator andwith a controller of vehicle functions; and a non-transitory computerreadable medium storing instructions which cause said processor: toidentify, via said sensor, a location of a first object along said grip,to illuminate, via said illuminator, a portion of said grip, adjacent tothe identified location, to further identify, via said sensor, a secondobject being at the illuminated portion of said grip, and to activate,via the controller, a vehicle function in response to the second objectbeing at the illuminated portion of said grip.
 23. The user interface ofclaim 22 wherein the instructions cause said processor: to illuminate,via said illuminator, two portions of said grip, and to furtheridentify, via said sensor, the second object as being at one of theilluminated portions.
 24. The user interface of claim 23 wherein theinstructions cause said processor to activate, via the controller,different vehicle functions in response to the second object being atone of the illuminated portions, according to which of the illuminatedportions the second object is at.
 25. The user interface of claim 22wherein the instructions further cause said processor: to identify, viasaid sensor, that the first object has moved to a different locationalong said grip, and to move, via said illuminator, the illumination toa portion of said grip that is adjacent to the different location. 26.The user interface of claim 22 wherein the vehicle is an autonomousdrive enabled vehicle, wherein the controller controls autonomous drivefunctions, and wherein said processor activates, via the controller, anautonomous drive function in response to the second object being at theilluminated portion of said grip.
 27. A user interface method for avehicle having a steering wheel that comprises (i) a grip, (ii) a sensoroperable to detect objects at a plurality of locations along the grip,and (iii) an illuminator operable to illuminate different portions ofthe grip, the method comprising: identifying a location of a firstobject along the grip; illuminating a portion of the grip, adjacent tothe identified location; further identifying a second object being atthe illuminated portion of the grip; and, activating a vehicle functionin response to the second object being at the illuminated portion of thegrip.
 28. The user interface method of claim 27 wherein saidilluminating comprises illuminating two portions of the grip, andwherein said further identifying identifies the second object as beingat one of the illuminated portions.
 29. The user interface method ofclaim 28 wherein said activating activates different vehicle functionsin response to the second object being at one of the illuminatedportions, according to which of the illuminated portions the secondobject is at.
 30. The user interface method of claim 27 furthercomprising: identifying that the first object has moved to a differentlocation along the grip; and moving the illumination to a portion of thegrip that is adjacent to the different location.
 31. The user interfacemethod of claim 27 wherein the vehicle is an autonomous drive enabledvehicle, and wherein said activating activates an autonomous drivefunction in response to the second object being at the illuminatedportion of the grip.